cargo build intermittently fails on darwin platform
#10022
Comments
akscram
changed the title
cargo build intermittently fails on darwin platform
|
The same problem with SIGABRT simulated on Monterey x86_64. |
|
It looks like this issue is not reproduced on latest cargo & rustc versions. If I run cargo from nixpkgs, I see this error: Nixpkgs have Rust 1.55: But if I install toolchains with rustup and use them, everything's ok: (stable is But also if I install 1.55.0 with rustup, I don't get this error: |
|
Could this be related to a change in default user limits (file descriptors) from macOS 11 to macOS 12? Changing the file descriptor limit on macOS 12 to the default value of macOS 11 ( |
|
I can confirm, raising fd limit helps on my system as well: |
|
I've tried it but Execution under |
|
@afh, sure. I tried to set And by running the same |
ehuss
added
the
S-needs-info
|
FWIW macOS 12.0.1 on my M1 machine reports a ulimit of 4864, and on my x86-64 machine of 2560. In both cases this appears to be a limit inherited from Terminal.app, as the system default for processes spawned by launchd is 256. I’m pretty sure this was true for macOS 11 as well, I don’t think the system default has changed, just potentially whatever ulimit your terminal emulator is using. |
|
@lilyball Thanks for the information. I use Terminal.app also. It looks like bare Terminal.app has That said, shouldn't nix builds be isolated from such configs on the caller side? |
|
Looks like nix-daemon on both of my systems is setting the ulimit to 4096. ❯ nix-build -E 'with import <nixpkgs> {}; runCommandNoCC "foo" {} "ulimit -n"'
this derivation will be built:
/nix/store/a71y1s1g82pgwpamzh9fd1vc30h7hfx0-foo.drv
building '/nix/store/a71y1s1g82pgwpamzh9fd1vc30h7hfx0-foo.drv'...
4096 |
|
Possibly the SIGABRT is unrelated to user limits then. What could other causes for SIGABRT be? 🤔 |
|
The problem is that Rust/Cargo was compiled without option |
|
@georgik Why would that cause random SIGABRTs though? |
|
@lilyball Unfortunately the root cause is not clear yet, we have just observed that switching to jemalloc in our builds of rustc (to match the setting used by official rust-lang releases) resolves the issue. I have tried a few things to debug this. Here's the backtrace when rustc crashes: (from lldb -- path/to/broken/rustc /tmp/rust-bitfield/src/lib.rs)Valgrind doesn't run on macOS 12 yet, so I wasn't able to test with it. I have also tried enabling Guard Malloc ( Using malloc_info in lldb to figure out if the pointer passed to realloc was allocated or not, So it looks like the pointer was indeed allocated. Maybe that's a bug in libsystem's heap implementation that it flags this as I think the issue is not with cargo and not with rustc-as-a-program. It looks like this also affects other programs compiled with rustc for macOS, when the system allocator is used on macOS 12, for example rayon-rs/rayon#901 (comment). |
|
@igrr Can you try turning on malloc stack logging? I'm wondering what I'm also curious how "authoritative" |
|
@akscram and everyone else here, can you say how you installed rust? Was it via the official builds (rustup or manually)? Or was it through a package manager? |
|
@lilyball unfortunately the issue stops being reproducible when malloc stack logging ( @ehuss In my and @georgik 's case these were unofficial builds of rustc done in our fork with Xtensa architecture support. The build scripts we use are in this repository. Official builds work correctly in the same environment. Edit: one extra detail, the build which exhibits this issue was performed on macOS 10.15. |
|
I have opened rust-lang/rust#92173 on the rustc side for this issue, as the realloc failure is almost certainly not a cargo issue. I'm still curious about the discussion above about the descriptor limits. That seems unrelated, but I'd like to better understand if it is something that needs attention. |
|
@ehuss My impression is the file descriptor limits thing was a red herring. I don't believe the default limits changed with the macOS update. |
|
Ah, thanks for the update. I'm going to close this in favor of rust-lang/rust#92173. |
I had to disable Clippy on macOS because it fails with SIGABRT on Monterey. I found an ostensible workaround[1], but it didn't work for me. Since we're already running Clippy in many other CI jobs, it shouldn't matter much that we don't run it on macOS. 1. rust-lang/cargo#10022 (comment)
Problem
cargo buildintermittently fails for a project or its dependencies and subsequent re-builds succeed. For example,bitfielddoesn't have dependencies but this problem is observed for the project itself:Subsequent rebuilds sometimes don't help and some project continue failing.
Steps
Possible Solution(s)
No response
Notes
This problem is observed on:
Version
The text was updated successfully, but these errors were encountered: